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THE FIRST OF A SERIES OF STUDIES DESIGNED TO INVESTIGATE 
THE SUITABILITY OF INSTRUCTIONAL MATERIALS WITH REGARD TO 
LEVELS OF DIFFICULTY IS REPORTED. THE STUDY ATTEMPTED TO 
DETERMINE WHETHER SOME RANGE OF DIFFICULTY MAXIMIZED THE 
AMOUNT OF INFORMATION STUDENTS GAIN AS A CONSEQUENCE OF 
READING INSTRUCTIONAL MATERIALS. IN ORDER TO FORM 129 FAIRS 
ACCORDING TO READING ABILITY, STUDENTS RANGING FROM GRADE 3 
THROUGH GRADUATE LEVEL WERE GIVEN A CLOZE TEST. ONE MEMBER OF 
EACH FAIR WAS GIVEN A CLOZE READABILITY TEST OVER A PASSAGE 
TO DETERMINE THE DIFFICULTY OF THE PASSAGE FOR THAT FAIR. THE 
OTHER MEMBER OF THE FAIR DETERMINED HOW MUCH INFORMATION THAT 
FAIR GAINED BY READING THE PASSAGE. HE FIRST GUESSED THE 
ANSWERS TO A MULTIPLE CHOICE TEST AND THEN READ THE PASSAGE 
AND TOOK THE SAME TEST AGAIN. INFORMATION GAIN WAS DETERMINED 
BY SUBTRACTING THE SCORE OF HIS FIRST TEST FROM THE SCORE OF 
THE SECOND. IT APPEARED FROM THE DATA OBTAINED THAT IT MAY BE 
POSSIBLE TO ESTABLISH FAIRLY DEFINITE STANDARDS OF WHAT IS A 
SUITABLE PASSAGE FOR USE BY A CHILD. IT WAS FOUND THAT SCORES 
ON CLOZE TESTS BO NOT DEPEND SOLELY ON A SUBJECT’S PRIOR 
KNOWLEDGE OF THE CONTENT OF A PASSAGE. EIGHT REFERENCES ARE 
GIVEN. THIS PAPER WAS PRESENTED AT THE INTERNATIONAL READING 
ASSOCIATION CONFERENCE. (BOSTON, APRIL 24-27, 1968). (KJ) 
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Teachers are often admonished to rive the child instruction only from materials 
which are suitable for him. Thorndike (7) first suggested standards of suitability 
which have come to be adopted in roughly this form# Materials are said to be 
suitable for use in the child's independent, unsupervised study if he can correctly 
answer at least 90 per cent of the comprehension questions asked him about the 
materials after he has studied them. The materials are regarded as suitable for 
use in supervised instruction if he can answer 75 per cent or more of the questions. 
V/hen the child can answer fewer than 75 per cent of the questions, study of the 
materials is thought to result in frustrations which cause the child to learn 
negative attitudes toward instruction. Since a number of well known authors (see 
Betts (1), Bond and Tinker (2), and Harris (5)) advocate the use of these standards, 
presumably the standards are widely used by teachers. The problem is that neither 
Thorndike nor anyone known to the author has given either logical or empirical 
reasons why these standards should be accepted. 

This problem is potentially too serious to be left unattended. To illustrate, 
materials might be difficult just because they contain information which is new to 
a reader. Barring a child from reading difficult materials might actually bar him 
from studying the very materials from which he would learn the most, thus creating 
boredom by forcing him to read dull, repetitious materials. Conversely, until we 
have definite evidence to the contrary, we cannot be certain that even materials 
on which children can answer 90 per cent of the questions are easy enough to pre- 
vent children from becoming frustrated and learning the negative attitudes which 
cause them to dislike and reject study. 

This paper reports the first of a series of studies designed to investigate 
this problem. The present study asks simply if there is some range of difficulty 
which maximizes the amount of information students gain as a consequence of reading 
instructional materials. Other studies the author currently has under way are 
investigating the difficulty levels at which student interest is maximized. 

Procedure 

The general pattern of this study involved these operations. First, students 
were given a cloze test in order to form pairs of students matched in reading ability^ 
Second, one member of each pair was given a cloze readability test over a passage to 
determine the difficulty of the passage for that pair of students. Third, the other 
member of the pair was used to determine how much information that pair gained by 
reading the passage. He first tried to guess the answers for a multiple choice test 
made from the passage and then later read the passage and took the same test again. 
Information gain was determined by subtracting his first score from the score he 
made the second time he took the test. The information gained by a pair of students 
was then plotted against the cloze difficulty of that passage for that pair of 
students to determine the shape of the curve. 



* The author wishes to acknowledge the long hours of labor Mr. Paul Berger of the 
University of Minnesota contributed to constructing the materials and gathering the 
data for this study. 







The cloze test used to match the pairs of students was made from a 263 word 
passage taken from an elementary psychology textbook by Cretch and Crutchfield (6)« 

A 52 item cloze test was made from this passage by deleting every fifth word from 
the passage and replacing the deleted words with underlined blanks of 15 typewriter 
spaces. The students were told how the test was made and that their task was to 
figure out what word was left out of each blank and to write their vresponse in the 

blank. Responses were scored correct only when they exactly matched the words 

deleted, ignoring misspellings which did not otherwise make the response ambiguous. 
The students were then ranked from high to low on the basis of their scores and 
then divided into pairs taking each successive pair of students and randomly assignir 
a member to each of the two groups designated hereafter as groups Y and Z. 

A cloze readability test and a multiple choice comprehension test was made for 
each of two passages designated passages A and B. Passage A contained 469 words 
and passage 3 contained 39Q words. Each passage descried in non-technical 
language a psychological experiment. Five cloze readability test forms were made 
for each passa 9 e by deleting words ll^ etc»j> to irtake the first form^ words f 

7, 12, etc., to make the the second, and so on until all of the five forms 

possible had been made. These forms were later randomly assigned to subjects, so 

that all five forms were used egually often. The multiple choice tests made from 
passage A contained 34 items and the one made from passage B contained 39. Each 
item contained four alternative responses. The comprehension tests underwent three 
editorial revisions each time trying them out on small groups of students. When 
these tests were scored, the scores were corrected for guessing by subtracting one- 
third the number of incorrect responses from the number correct. The rcHabilities 
of the cloze readability tests, pooling all five forms in a single split-half 
correlation, were .92 and .89 for passages A and B, respectively. The reliabilities 
of the comprehension tests for passages A and B were .04 and .06, respectively. 

Both sets of reliabilities were calculated using the data reported in this study, 
the scores from the second administration of the comprehension test being the ones 
used in that case. All reliability correlations were corrected for test length. 

This study was designed to partially replicate its findings and to make maximum 
use of student testing time by using two different sets of passages and tests but 
the same set of students. Table 1 shows the order in which the tests were admin- 
istered. The test sessions were always at least a week apart with none spaced more 

Table 1 

Design of Test Administration Schedule 






Vest Session 



1 



2 



3 



K 

V 



Matching Test 
Matching Test 



Comprehension A and Cloze B 
Comprehension B and Cloze A 



Read A and Comprehension A 
Read B and Comprehension B 









two weeks apart. Ao time limit was imposed on any of the tests. Using tms 
testing design, the cloze score of member X of each pair v-^as used to determine tne 
cloze difficulty of passage « for his pair while the tvio comprehension scores of 
member Y were used to determine the information that pair gained as a consequence 
of reading oassage Conversely, the cloze score of member Y was used to deter- 
mine the cloze difficulty of passage A for that pair and the comprehension scores 
of member )( vjere used to determine the information the pair gained from reacing 

passage A. 



Originally it was planned to use just second year junior college stuoents ^ 
in the study, dowever, it proved necessary to use student;, ranging ^ 

through graduate level in order to obtain a sufficient range of cloze dir^culty 
scores. A total of 130 pairs of students were tested, 25 pairs in grade 
in grade 5, 15 in grade 1, 2C in grade 11, 24 in junior college, and 15 pairs 
enrolled in a graduate course, 'because of absences, the data presente *or 
passage A are based on 129 pairs and those for passage 3 on 125 pairs. 

Analysis and Results 



All scores viere converted to percentage scores and then the cloze divricul ..y 
score for each pair was correlated with the information gam score. ^ Third degree 
polynomial curves were fit to the data yielding a multiple correlation .o. on 
passage and a correlation of .52 on passage 3. The multiple correlations were 
calculated by determining the multiple correlations between the information gam 
scores and the cloze difficulty scores, their squares, and their cubes. Only the 
linear and quadratic terms contributed significant amounts of variance to these 
correlations. 



In order to determine the shapes of the curves relating information gam to 
cloze difficulty, eighth degree polynomial curves were fit to botn tne data from 
passage A and passage 3. These curves were highly similar having a 
.95 over the range of observed scores. In order to show the general shape o» toe 
relationship between information gain and cloze difficulty the two sets or data 
were combined and a single eighth degree polynomial curve rit^to the data, ^is 
curve is shown in Figure 1. The curve shows that the pairs or students who made 
scores of less than 17 per cent on cloze readability tests on tne passages 
tended to exhibit little gain in information when they read the passages. But for 
pairs making cloze readability scores in the range between 17 and 37 
there was a sharp increase in the information gamed. As pairs made 
cloze readability scores, their information gam scores tenoed to level oir with, 
perhaps, come slight tendency to increase. 



In order to assess the fit of this curve to the data, the scores were divided 
into l4 intervals on the cloze scale and the mean gain scores calculated at each 
interval. Tgbl(&2 shows these data* Mote that nearly all of these means fall 
close to the curve, .'lone of the differences betvjeen pairs of interval means from 
the two different passages vias significant when t. « tests were appHed._^ 
sat of means tend to be consistently higher than the other set. .Ihile u oi^ the 
14 differences favored passage 3, this result, vjhen the sign 1^®^^ was applied, v/as 
not significant and there appeared to be no consistent tenancy for the 
between the curves to concentrate just in one segetnent of the curve, 
two different sets of materials seemed to yield roughly i dent i cal results. 
the means showed fairly large differences, the number or subjects on which the means 
were based tended to be small suggesting that the differences were due to this 
source of instability. 
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Table 2 

Mean of the Information Gain Scores in Each 
Interval of the Cloze Difficulty Scores 



Mean Gain Score 



Cloze 

Diff icul ty 
Interval 



Passage A 



Passage B 



0 - ^ 
5-9 

10 - 14 
15 - 19 
20 - 24 
25 - 29 
30 - 34 
35 - 39 
40 44 

45 - 49 
50 « 54 
55 « 59 
60 - 64 

65 - 69 



Number 


Mean 


Number 


Mean 


7 


.022 


3 


.024 


9 


.123 


4 


.121 


10 


.002 


9 


.149 


’ !0 


.005 


9 


c 1 00 


8 


.107 


5 


.170 


3 


.2!l3 


19 


#210 


13 


.304 


11 


.389 


5 


.181 


14 


.442 


5 


.474 


9 


.429 


15 


.407 


10 


.451 


16 


.422 


20 


.439 


12 


.426 


6 


.519 


12 


.442 


4 


.354 


4 


.472 


2 


.470 



Discussion 

It appears from these data that it may be possible to establish 
definite standards of what is or is not a passage suitable for use by a child. 

These data seem to show that a student can gain very little information from 
studying materials on which his cloze readability score is below 37 per cent and 
that using materials much easier than the 37 per cent level will permit *he child 
to acquire only slightly more knowledge while reading them# However, these J[®suits 
should be regarded as preliminary. Only two passages were used and both of those 
were quite similar in difficulty. Further, it is not certain how well these results 
will generalize to comprehension tests made by other test writers# 

The question of whether the traditional 75 and 90 per cent levels 
the beginning of this report are adequate can be only partially answered from these 
daLo Bormuth (3 and 4) found that the cloze readability test scores t^^ 

these two levels were ^^4 and 57 per cent, respectively# These levels nt/ 

ure ! by the broken vertical lines. It can be seen from this ^^^ure that s^^nts 
who are given instruction from materials at one of these levels gain little 
information than students given instruction from materials at the other level# And 
both levels seem to result in near maximum information gam. 

However, the question of whether the 44 and 57 per cent cloze - 

criteria are desirable cannot be resolved without finding out how the difficulty of 
matLials influences students' interests and attitudes toward 
mav be that the or. 75 per cent criterion is too stringent and that materials 
having cloze or comprehension test readabilities as low 37 or 65 per cent, 
resoectively, may be perfectly suitable for students when both interest and infer- 
mS gain ar^Ln! idered. Simitarly, there is presently no way to be sure that 
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passacies on wln'ch a student score*, above the cloze and comprehension test 
readability criterion scores of 57 and 50 per cent, respectively, are no so 
simole to the student that he rejects them as being insipid and an insult ^ 

intelligence. Finally, there is no reason to think that the '--I*®*;"" 
necessarily be the same for students of all ages, or. passages at all levels 
difficulty, and on passages on all topics. Obviously this topic calls for a 
great deal more research. 



On Cloze Text Theory 



These data are al«?o relevant to the question of whether cl( 
the redundancy of the passages from which they are made an v 

... .( _... j I Hi iSivamnlp- acRert ' 



loze tests measure 

• • 

the redundancy of the passages rrom wmcn uiey a. » 

cloze test theory, '.'eaver and Kingston (0) for example ® ®hev 

are largely a measure of redundancy but fail to make clear m just what 
are using the term redundancy. In its most rigorous ur 

frequencies with which patterns of letters, words, parts or speech and ° 

in the lanouaoe. It is a logical absurdity to assert that cloze tests measure tne 
frequencies of occurrence of anything except the subject's responses. ^ 

may^ave meant was that cloze tests measure either / P*’?* ^a^ 

of the information contained in a passage or tne vamaliarity ^ 

terns of the subject. The oresent study shows rather decisively that scores on 
ctoze ?L^s rnTd.;pend solely upon a%ubjecfs prior knowledge of the content 
of a oassage. ATcloze scores increased, information gam increased. Further, 
scores on the comprehension test administered before the subjects ha ® 

passage had correlations of .0$ and .11 with cloze scot®®* scored do 

Lre not significantly different from zero at the .05 level. If cloze scores do 
reflect the^subject's^prior knowledge of the information in the passages, the 
effect seemed so weak as to be negligible in the present study. 
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